System and method for dynamic cooperative distributed execution of computer tasks without a centralized controller

ABSTRACT

A system and method is provided for cooperative execution of distributed tasks by networked computers without the use of a centralized controller to coordinate the task execution. Each computer has an execution agent that cooperates with the execution agents of the other computers to carry out the execution of a given sequence of tasks. The execution instructions for the task sequence are provided to a first computer in the selected set of computers for task execution. The first computer processes the instructions and forwards them to its peer computers so that each of them knows the tasks it is to perform in connection with the tasks of the other computers. The computers then executes the tasks assigned to them and use peer-to-peer communications to provide status update to their peer computers to synchronize and coordinate the task execution.

TECHNICAL FIELD

This invention relates generally to the controlled execution of tasksover a computer network, and more particularly to a computer frameworkfor cooperative automated execution of distributed tasks by a set ofcomputers without a centralized controller.

BACKGROUND OF THE INVENTION

Computer software and hardware operations, especially those that involvecomplex interactions between different computers such as thosesupporting various networking protocols and distributed transactions,often have to be extensively tested to ensure that they functionproperly. Typically, to test a given type of computer interactiveoperation, a set of test computers are chosen to run a test case thatincludes a sequence of tasks to be performed by different ones of thetest computers in an interactive fashion. In one conventional testframework, the execution of the task sequence of the test run iscontrolled by a centralized test controller. The test control goesthrough the sequence of tasks one by one, instructs a corresponding testcomputer to carry out one task, receives the result of the task, andinstructs another test computer to carry out the next task in thesequence based on the outcome of the previous task.

The use of a centralized controller to control the automated executionof distributed computer tasks, however, places a significant constrainton the availability of test computers. To enable the centralizedcontrol, each test computer is required to form a communication linkwith the centralized test controller to receive instructions and reporttask execution status and results. This requirement excludes manycomputers that do not have the capability of forming a direct networkconnection with the controller from being used for distributed computertesting.

SUMMARY OF THE INVENTION

In view of the foregoing, the present invention provides a framework forautomated dynamic execution of distributed tasks by a set of computerswithout the need to use a centralized controller to coordinate theexecution of the tasks. Although this framework is especiallyadvantageous for the application of testing interactive computeroperations, it can also be used to carry out other types of distributedtasks. In accordance with the invention, the execution of a sequence oftasks is coordinated through the cooperation of the peer computersselected to perform the tasks. Each computer has an execution agent forhandling the cooperative execution of the tasks. Task executioninstructions that identify the sequence of tasks to be executed and thecorresponding computers that are to perform the individual tasks aregiven to a first computer in the set of selected computers. The firstcomputer passes the task execution information on to the other peercomputers involved in the distributed execution so that each computerknows which tasks it is to perform in relation to the tasks of the othercomputers. The execution agents of the computers then communicate witheach other for status update and synchronize to execute the sequence oftasks.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram generally illustrating an exemplary computersystem on which an embodiment of an execution agent for cooperativeexecution of distributed tasks in accordance with the invention may beimplemented;

FIG. 2 is a schematic diagram showing an embodiment of a system thatincludes a set of computers for cooperative distributed execution oftasks of a test run;

FIG. 3 is a flow diagram summarizing the operation of execution agentson selected computers in an embodiment of the invention for automatedexecution of distributed tasks without a centralized controller; and

FIG. 4 is a schematic diagram showing the structure of execution agentson the computers in the system of FIG. 2 for carrying out thecoordinated execution of distributed tasks by the computers.

DETAILED DESCRIPTION OF THE INVENTION

Turning to the drawings, wherein like reference numerals refer to likeelements, the invention is illustrated as being implemented in asuitable computing environment. Although not required, the inventionwill be described in the general context of computer-executableinstructions, such as program modules, being executed by a personalcomputer. Generally, program modules include routines, programs,objects, components, data structures, etc. that perform particular tasksor implement particular abstract data types. Moreover, those skilled inthe art will appreciate that the invention may be practiced with othercomputer system configurations, including hand-held devices,multi-processor systems, microprocessor-based or programmable consumerelectronics, network PCs, minicomputers, mainframe computers, and thelike. The invention may be practiced in distributed computingenvironments where tasks are performed by remote processing devices thatare linked through a communications network. In a distributed computingenvironment, program modules may be located in both local and remotememory storage devices.

The following description begins with a description of a general-purposecomputing device that may implement components of a framework of theinvention for coordinated execution of distributed tasks by a selectedset of computing devices. The framework of the invention for cooperativedistributed task execution without using a centralized controller willbe described in greater detail with reference to FIGS. 2-4. Turning nowto FIG. 1, a general purpose computing device is shown in the form of aconventional personal computer 20, including a processing unit 21, asystem memory 22, and a system bus 23 that couples various systemcomponents including the system memory to the processing unit 21. Thesystem bus 23 may be any of several types of bus structures including amemory bus or memory controller, a peripheral bus, and a local bus usingany of a variety of bus architectures. The system memory includes readonly memory (ROM) 24 and random access memory (RAM) 25. A basicinput/output system (BIOS) 26, containing the basic routines that helpto transfer information between elements within the personal computer20, such as during start-up, is stored in ROM 24. The personal computer20 further includes a hard disk drive 27 for reading from and writing toa hard disk 60, a magnetic disk drive 28 for reading from or writing toa removable magnetic disk 29, and an optical disk drive 30 for readingfrom or writing to a removable optical disk 31 such as a CD ROM or otheroptical media.

The hard disk drive 27, magnetic disk drive 28, and optical disk drive30 are connected to the system bus 23 by a hard disk drive interface 32,a magnetic disk drive interface 33, and an optical disk drive interface34, respectively. The drives and their associated computer-readablemedia provide nonvolatile storage of computer readable instructions,data structures, program modules and other data for the personalcomputer 20. Although the exemplary environment described herein employsa hard disk 60, a removable magnetic disk 29, and a removable opticaldisk 31, it will be appreciated by those skilled in the art that othertypes of computer readable media which can store data that is accessibleby a computer, such as magnetic cassettes, flash memory cards, digitalvideo disks, Bernoulli cartridges, random access memories, read onlymemories, storage area networks, and the like may also be used in theexemplary operating environment.

A number of program modules may be stored on the hard disk 60, magneticdisk 29, optical disk 31, ROM 24 or RAM 25, including an operatingsystem 35, one or more applications programs 36, other program modules37, and program data 38. A user may enter commands and information intothe personal computer 20 through input devices such as a keyboard 40 anda pointing device 42. Other input devices (not shown) may include amicrophone, joystick, game pad, satellite dish, scanner, or the like.These and other input devices are often connected to the processing unit21 through a serial port interface 46 that is coupled to the system bus,but may be connected by other interfaces, such as a parallel port, gameport or a universal serial bus (USB) or a network interface card. Amonitor 47 or other type of display device is also connected to thesystem bus 23 via an interface, such as a video adapter 48. In additionto the monitor, personal computers typically include other peripheraloutput devices, not shown, such as speakers and printers.

The personal computer 20 may operate in a networked environment usinglogical connections to one or more remote computers, such as a remotecomputer 49. The remote computer 49 may be another personal computer, aserver, a router, a network PC, a peer device or other common networknode, and typically includes many or all of the elements described aboverelative to the personal computer 20, although only a memory storagedevice 50 has been illustrated in FIG. 1. The logical connectionsdepicted in FIG. 1 include a local area network (LAN) 51 and a wide areanetwork (WAN) 52. Such networking environments are commonplace inoffices, enterprise-wide computer networks, intranets and the Internet.

When used in a LAN networking environment, the personal computer 20 isconnected to the local network 51 through a network interface or adapter53. When used in a WAN networking environment, the personal computer 20typically includes a modem 54 or other means for establishingcommunications over the WAN 52. The modem 54, which may be internal orexternal, is connected to the system bus 23 via the serial portinterface 46. In a networked environment, program modules depictedrelative to the personal computer 20, or portions thereof, may be storedin the remote memory storage device. It will be appreciated that thenetwork connections shown are exemplary and other means of establishinga communications link between the computers may be used.

In the description that follows, the invention will be described withreference to acts and symbolic representations of operations that areperformed by one or more computers, unless indicated otherwise. As such,it will be understood that such acts and operations, which are at timesreferred to as being computer-executed, include the manipulation by theprocessing unit of the computer of electrical signals representing datain a structured form. This manipulation transforms the data or maintainsit at locations in the memory system of the computer, which reconfiguresor otherwise alters the operation of the computer in a manner wellunderstood by those skilled in the art. The data structures where datais maintained are physical locations of the memory that have particularproperties defined by the format of the data. However, while theinvention is being described in the foregoing context, it is not meantto be limiting as those of skill in the art will appreciate that variousones of the acts and operations described hereinafter may also beimplemented in hardware.

Referring now to FIG. 2, the present invention is directed to a newframework 70 for automated execution of distributed tasks by a selectedset of computers. In accordance with the invention, the sequence ofdistributed tasks is executed by the corresponding computers in anorderly fashion through the cooperative interactions among thecomputers. As a result, there is no need to use a centralized controllerto coordinate and control the progress of the execution of the tasks.For simplicity of illustration, FIG. 2 shows only three computers 80, 82and 86 in the machine set 88 for executing a given sequence ofdistributed tasks. It will be appreciated, however, that the number ofcomputers required for carrying out a sequence of tasks depends on theparticular sequence.

In a preferred embodiment as described below with reference to FIG. 2,the distributed task execution framework of the invention is used forthe application of automated computer testing, and the followingdescription will refer to the context of executing tasks involved in acomputer test run. It will be appreciated, however, that the cooperativeexecution of distributed tasks of the invention is not limited totesting applications, and can be used equally well for executing othertypes of distributed tasks.

To carry out the distributed tasks without the help of a centralizedcontroller, each of the computers 80, 82, 86 participating in the taskexecution is provided with an execution agent (EA) 90, 92 or 96. Theexecution agent 90 is responsible for launching the assigned tasks onthe test machine 80 on which it resides, and for communicating with theexecution agents 92, 96 on the other test machines 82, 86 to coordinatethe execution of the distributed tasks of a test run. In one embodiment,the execution agent is implemented as part of a dynamic link library(.dll) for computer testing. The functionality of the execution agentswill be described in greater detail below.

To facilitate the description of a preferred embodiment for computertesting operations, some basic concepts regarding automated execution byselected machines as used in the embodiment are first defined.Generally, a “task” is the smallest unit of execution by a machine. A“job” is a list of tasks, and a “run” is a list of jobs. A set ofmachines is reserved for a run. All machines participate in theexecution of each job in the run even if they do not have any tasks ofthe job or even if that job's logical-to-physical machine mapping doesnot include that particular physical machine.

As implemented in the embodiment, there are three different types oftasks. An “Exe” task is a command line (anything that can be created asa process) to be executed. A “Copyfile” task copies a list of files froma source to a destination. A “RunJob” task has a pointer to another job.In this context, the job being pointed to is called a “subjob,” whichwill be explained in greater detail below. The execution of the RunJobtask is basically the execution of the subjob. When the execution agentencounters this task, it finds the associated subjob and executes it,and the task's pass/fail status is determined by the success/failure orthe subjob. The common properties for all types of tasks include theuser name under which the task is run, the password for the user name,the domain for the user credentials, and a FailureAction property thatspecifies the action to be taken by the execution agent if the taskfails. Each task also has a TaskCategory property, which tells theexecution agent what kind of task it is. This property in combinationwith the FailureAction property determines the course of action to betaken upon the failure of the task. Also, one of the common propertiesof the task is the “Logical Machine” to which the task is assigned. TheMachineName, which is the name of the physical machine that is toactually execute the task, is “found out” by using the LogicalMachineand the LogicalMachine To Physical Machine Mapping.

In one implementation, there are five different categories of tasks:SetupJob, Regular, Cleanup, CleanupPass, and CleanupFail. The SetupJobtasks are precursors to the tasks that actually run the tests and mayinclude any task that is needed to set up the machine, which can varyfrom copying source binaries to deploying an operating system. TheRegular tasks are the tasks that actually run the tests. The Cleanuptasks are involved in any cleanup operation that needs to be doneregardless of the success/failure of the Setup and Regular tasks. TheCleanupPass tasks are involved in any cleanup operation that needs to bedone upon the non-failure of the Setup and Regular tasks. In contrast,the CleanupFailure tasks are those that are involved in any cleanupoperation to be done upon the failure of the job. Each task in any ofthe five categories can be of any type (Exe, Copy, or RunJob). Asmentioned above, a “job” is a list of tasks. The tasks in a job maybelong to different categories. Within the job the tasks are executed inthe following order: Setup, Regular, Cleanup, and then CleanupPass orCleanupFail, depending on the status of the job at the end of theregular tasks.

In accordance with a feature of a preferred embodiment, a task may bethe execution of a “subjob.” A subjob is basically the same as a jobexcept for a few differences. A subjob has a “parent” task (a RunJobtask) that refers to it, and the subjob is executed as part of runningthe parent task within some other job, which is referred to as theParentJob in this context. When the list of tasks of a given test run isgiven to the test machines, a subjob may be referred to withoutspecifying on which physical machines the subjob is to run. In thatcase, the subjob will be executed on the machine on which the parenttask is supposed to run. On the other hand, if a mapping to a physicalmachine is specified for a subjob, the subjob will be executed on thatmachine.

Generally, when a sequence of tasks is given to a set of test computers,the execution instructions may identify the assignment of those taskswith respect to a set of logical machines, and provide a mapping betweenthe logic machine set to the physical machines selected to carry out thetasks. For instance, the instructions may state that task number 5 is tobe performed by logical machine A, which is mapped to physical machinesX and Y. In that case, further splitting of the tasks with respect tothe physical machines may be required. In one embodiment, if the logicalmachine a task is supposed to run is mapped into two or more physicalmachines, the task is split into as many tasks to have one task perphysical machine. The splitting is achieved by making a copy of the taskand changing the MachineName property of the task to the name of thephysical machine that will run the task. If the task is a RunJob taskthat refers to a subjob with mapping, the task is not split, and theMachineName of the task is changes to the list of machine namesspecified by the subjob's mapping. If the RunJob task does not specifythe logical-to-physical machine mapping for the subjob, then the subjobis also split with one subjob per task. The task dependencies areadjusted accordingly after the splitting of the tasks

To use the set 88 of test machines to execute the sequence of tasks fora test run, task execution instructions are provided to one of the testmachines. In the illustration of FIG. 2, the computer 80 is used forthat purpose. The task execution instructions are generated by a testmanagement server 100, which includes a test case database 102 andscheduler 106. The scheduler 106 takes a test run and finds a set 88 ofavailable test machines for running the test case. The test managementserver 100 further includes a push daemon 108 for delivering the taskexecution instructions to the group of test computers. If a networkconnection 118 is formed between the push daemon 108 and the computer80, the push daemon may send the execution instructions to the computer80 via the network connection. Alternatively if there is no networkconnection available between the test management server 100 and any ofthe test computers 80, 82 and 86, the execution instructions may beloaded onto the computer 80 using a transportable medium, such as aCD-ROM. The execution instructions identify the tasks and the order inwhich they are to be executed, and which test computer is to performwhich task. In other words, the execution instructions provideinformation for each test computer involved in the test run to know whatits tasks are in relation to the tasks of the other computers. Inaccordance with a feature of a preferred embodiment, the executioninstructions are provided in the form of an XML file or document. TheXML file allows easy viewing and editing and provides a standard formatfor the execution instructions to be coded.

Referring now to FIGS. 2 and 3, to initiate a test run, the Push Daemon108 of the test management server 100 sends a Run XML file 120 thatcontains the execution instructions for a test run to the executionagent 90 on the computer 80 in the set of computers selected forexecuting the test run (step 122). When the execution agent 90 of thefirst computer 80 receives the Run XML document, it first checks to seeif it is already running any other test run (step 126). If so, itrejects the message and exits the run process. If it is not running anyother test, it calls a EaCreateRunTimeXML API function to transform theRun XML (step 128). This transformation involves splitting the tasksspecified in the Run XML document, adjusting the dependencies of thetasks, and adjusting the subjobs, if any. As part of this reformattingoperation, the execution agent splits the Run XML into several smallerXML files, including one XML file for each job of the test run and aglobal JobList XML file that contains a list of all jogs of the testrun. It also forms a concatenated version of all of these XML files thatis called a “PeerRun XML.” The execution agent on the first machine thenidentify the other machines involved in the run by reading the list ofthe test machines from the Global JobList XML, and sends the PeerRun XMLfile to each of the other machines (step 130). When the execution agenton a peer machine 82 receives the PeerRun XML 84, it recognizes thedocument to be from the execution agent of the first machine. If thepeer machine is already involved in another test run (step 132), itsexecution agent 92 returns an error message to the first machine 80(step 136). Otherwise the execution agent of the peer machine splits theconcatenated XML files and store them in the memory (step 138). Onceevery machine involved in the test run receives the executioninstructions in the XML files, each test machine starts running the testjobs one by one sequentially according to the test XML files (step 140)and sends the results of the test jobs to the test result database 72 Inexecuting the tasks of the test run, the execution agent of each testmachine communicate with all the execution agents of the other testmachines by sending messages 104 that provide status updates to everyone(step 144). They also synchronize with each other at the beginning ofevery run or job to make sure that all the machines start at the sametime. If the machines do not have direct network connections to the testresult database 72 of the test manager server, the test results may besent to a local database 74. Once all the jobs of the test run arecompleted, the test results collected by the local database 74 may besent to the database 72 of the test management server 100 for diagnosesand reporting.

In a preferred embodiment, the execution agent has several importanthigh-level features. First, the execution agent can execute a jobinvolving multiple peer test machines. It accomplishes this by usingpeer-to-peer messaging, whereby the execution agents running all thetest machines synchronize with each other and receive the up-to-datestatus of what is happening on all the other test machines.

During the course of executing a job, a test machine may be taken offthe network. The execution agent is programmed to support such ascenario and will resume communications with the other machines when itsmachine comes back online. Jobs can also span over reboots. To handlethat scenario, the execution agent stores its state information on thefile system of the local machine and will recover its state when themachine reboots and resume execution. The execution agent also supportsthe installation of an operating system on the local machine as part ofthe test job. If the machine has multiple partitions, it also supportsfresh installations of the operating system with formatting anddeploying an image in the machine. In that case, the execution agentwill continue the execution of the test run tasks after the newoperating system boots up on the machine.

To ensure the reliability of the test results, the execution agent ispreferably built such that any kind of failure is either recovered orreported for follow-up. If one of the test machines for a job is hung orout of reach for some reason, the execution agents on the other machinesdetect it and cancel the current execution and report a failure. It alsotakes care of hooking into the debugger for both user mode and kernelmode breaks on the machines.

The execution agent also provides functionality to collect test andsystem information after the execution of the job. Different informationgathering commands can be executed based on the outcome of the job. Thejob can request for minimal logs to be gathered if the job passes andfor detailed logs to be captured if the job fails. Failure actions aresupported by the execution agent. If any task in the job fails, the jobcan request for many different kinds of actions to be taken. The job canbe canceled, and a set of failure commands can be executed to collectinformation about the system and the job for later diagnosis. The jobcan also be defined to recover the machine back to a good state if thefailure leaves the test machine in a bad state. In this case, themachines are recovered and returned to the available test machine poolso that new test jobs can be assigned to this machine.

Tests can be defined in great detail. Apart from giving the actualcommand line to be executed for an executable task, the user can alsospecify the credentials of which user account to run the task under. Theuser can also specify which computer is to be used as a monitor forshowing the process status or to create a new monitor session for thetask. The execution agent also supports the ability for any execution tobe canceled at any point of time. The cancel request is processed andpassed on to all the peer machines, and the cleanup processes areexecuted. The machines are then returned back to the free machine poolfor further job assignments. The execution agent is incorporated witherror logging to the database for any kind of failures so that allinformation is available from the user interface for the user todiagnose the issues.

Turning now to FIG. 4, in one implementation, the execution agent 90 hasseveral major components. The Run object handles a test run. If the runis a multi-machine operation, and the machine on which the executionagent resides is the first machine to be called with the Run XML, theRun object 150 processes the Run XML to split it and produce a series ofsmaller XML files to work with, and sends the concatenation of all theprocessed XML files to the other machines involved in the test run. Onthe other hand, if the local machine is not the first machine as in thecase of the computer 82 and is instead receiving the concatenated XML,the Run object 152 splits the concatenated XML and stores the files. TheRun object 150 or 152 then instantiates Job objects 154, one for eachjob present in the test run, and initialize them. The Job objects 154are grouped according to the execution phase they belong to (e.g.,Setup, Regular, and Cleanup). The Run object of each computerinitializes all common data by providing accessors for Job and Taskobjects to access them, and executes the jobs in a sequential orderaccording to the execution instructions in the XML files. During thetest run, the Run objects synchronizes the execution of each job in allthe machines such that at any point of time all the machines involvedrun the same job. The Run object of each computer also initializes andstarts an EA HB object 154 for maintaining a “heartbeat” with all thepeer machines. The run is canceled upon failure of heartbeat (i.e., thepeer machine being unresponsive) for three consecutive times. The Runobject is also responsible for sending updates to the peer machines andthe result database at the beginning and completion of the run.

The Job object 154 instantiates Task objects 160 for each task of therun that is to be performed by the local machine and initializes them.The Task objects 160 are grouped according to the execution phase theybelong to. The Job object 154 manages the dependency between the tasks,which involves executing only those tasks that do not have anydependency (i.e., pending the completion of another task) at that giventime, and at the completion of a task removing the dependencies fromother tasks that have dependency on this task. Generally, a job is amulti-machined entity, and there is on Job object in each test machinefor each job. All the Job objects on the test machines talk to eachother to coordinate the task execution and dependencies. Each Job objectsends updates to the peer machines and the database at the beginning andcompletion of the associated job.

The Task object 160 is responsible for executing an associated task in aspecified user context and terminal session. It has the ability tocancel or kill the task, if needed. The Task object 158 sends updates tothe peer machines and the database at the beginning and completion ofthe task.

The MessageDelivery object 162 is responsible for delivery of messagesto the peer machines. It gets the routing information from the Run XMLand constructs the headers to the peer machines and the database. Tothat end, it has to exposed functions: SendMessageToPeers andSendMessageToSource. It also maintains a list of the peer machines forreference.

The EaCreateRunTimeXML object 158 is called by the Run object 150 toprocess the input Run XML file and splits the file into various smallerXML files as described above. It also handles the mapping from logicalmachines to physical machines for the tasks and subjobs.

The EA SDK objects are a group of APIs that are used to interface withthe execution agent. The EaCreate function creates an instance of theexecution agent. It takes the Run XML as the input and returns a handlefor the execution agent. It is signaled on completion of the run. TheEaWaitForMultipleObjects is used to wait on the execution agent handles.The EaSignal function is used to signal the execution agent and tocancel the execution agent. The EaQueryStatus function, when invoked,returns the status of the run at that point of time in an XML format.The EaClose function closes the handle for the execution agent.

In view of the many possible embodiments to which the principles of thisinvention may be applied, it should be recognized that the embodimentsdescribed herein with respect to the drawing figures are meant to beillustrative only and should not be taken as limiting the scope of theinvention. Therefore, the invention as described herein contemplates allsuch embodiments as may come within the scope of the following claimsand equivalents thereof.

1-21. (canceled)
 22. A computer-readable medium havingcomputer-executable instructions for coordinated execution ofdistributed tasks comprising: receiving by a peer computer in a group ofat least one peer computers a set of execution instructions for the peercomputer, the set of execution instructions identifying a sequence oftasks to be performed by the peer computer; executing by the peercomputer the tasks identified by the set of execution instructions inconnection with execution of any tasks assigned to other peer computersin the group of at least one peer computers; transmitting by the peercomputer to any other peer computers in the group of at least one peercomputer communication messages containing task execution status tosynchronize and coordinate the execution of the tasks.
 23. Thecomputer-readable medium of claim 22 wherein the sequence of tasks to beperformed constitutes a test run of interactive computer operations. 24.The computer-readable medium of claim 22 wherein the instructionsinclude a job for executing a predefined set of tasks.
 25. Thecomputer-readable medium of claim 22 further comprising executioninstructions provided in an input XML document.
 26. Thecomputer-readable medium of claim 22 having further computer-executableinstructions for the peer computer to report results of execution of thetasks to a database.
 27. A method of performing coordinated execution ofdistributed tasks by a group of peer computers, comprising: receiving bya peer computer in a group of at least one peer computer, a set ofexecution instructions for the peer computer, the set of executioninstructions identifying a sequence of tasks to be performed by the peercomputer; executing by the peer computer the tasks identified by the setof execution instructions in connection with execution of any tasksassigned to other peer computers in the group of at least one peercomputers; transmitting by the peer computer to any other peer computersin the group of at least one peer computer communication messagescontaining task execution status to synchronize and coordinate theexecution of the tasks.
 28. The method of claim 27 wherein the sequenceof tasks to be performed constitutes a test run of interactive computeroperations.
 29. The method of claim 27 wherein the executioninstructions include a job that executes a predefined set of tasks. 30.The method of claim 27 wherein the execution instructions are providedto the peer computer in an input XML document.
 31. The method of claim27 further comprising reporting results of execution of the tasks by thegroup of at least one peer computer to a database.
 32. A computingdevice for performing automated execution of distributed tasks, thecomputing device comprising, a peer computer, the peer computerconnected by a communication link to at least one other peer computer;at least one execution agent residing on the peer computer, theexecution agent being programmed for receiving a set of executioninstructions for the peer computer, the set of execution instructionsidentifying a sequence of tasks to be performed; the execution agentcapable of executing tasks assigned to the peer computer and the peercomputer transmitting to the other peer computers communication messagescontaining task execution status to synchronize and coordinate theexecution of the sequence of tasks.
 33. The computing device of claim 32wherein the sequence of tasks to be performed constitutes a test run ofinteractive computer operations.
 34. The computing device of claim 32wherein the execution instructions include a job for executing apredefined set of tasks.
 35. The computing device of claim 32 whereinthe execution instructions are in an input XML document.
 36. Thecomputing device of claim 35 wherein the execution agent of the peercomputer is further programmed to process the input XML document toderive the execution instructions for sending to other peer computers.37. The computing device of claim 36 wherein the execution agent of thepeer computer is further programmed to format the execution instructionsas a second XML document.
 38. The computing device of claim 32 furtherincluding a test result database, and wherein the execution agent of thepeer computer is further programmed for reporting results of theexecution of tasks to the test result database.